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Abstract. We describe a generating tree approach to the enumeration and exhaustive generation of fc-non- 
nesting set partitions and permutations. Unlike previous work in the literature using the connections of 
f**^ ' these objects to Young tableaux and restricted lattice walks, our approach deals directly with partition and 

' permutation diagrams. We provide explicit functional equations for the generating functions, with fc as a 

parameter. 

OiJ' 
< 

0\ ' 1. Introduction 

(N . 

Arc annotated sequences, also called arc diagrams, are structures that have proved particularly useful for 
encoding a variety of combinatorial classes. An arc annotated sequence is a row of increasingly labelled 
vertices, from 1 to n, with some collection of arcs between them, restricted according to the object. Many 
■ combinatorial objects, including matchings, set partitions, labelled graphs, RNA substructures and per- 

fH I mutations, have straightforward descriptions using arc annotated sequences [6, 7, 8, 9, 11]. Two important 

patterns that occur in arc annotated sequences, and thus can be defined for each of the combinatorial objects 
that these sequences represent, are nestings and crossings. A set of k arcs forms a k-nesting (respectively 
k-crossing) if each of the (2) pairs of arcs nest (resp. cross). Figure 1 illustrates a 3-nesting and a 3- 
crossing. Precise definitions of /c-nestings in set partitions and permutations are given in Sections 2.4 and 5, 
respectively. 

in 

00 ■ Figure 1 . A 3-nesting and a 3-crossing. 

o . 

A diagram is k-nonnesting if it contains no fc-nesting, and it is k-noncrossing if it contains no /c-crossing. 
Chen, Deng, Du, Stanley and Yan [6] proved that fc-nonnesting partitions are equinumerous to fc-noncrossing 
partitions bijectively using Young tableaux as an intermediate object. In fact, they proved stronger distri- 
, bution results, both for partitions and for matchings. This work was extended by de Mier [10] to show that 

^ ' for embedded labelled graphs with certain restrictions, the number of fc-noncrossing ones is the same as 

5^ , the number of fc-nonnesting ones. Most recently, Burrill, Mishna and Post [5] proved analogous results for 

permutations. 

For most classes described by an arc annotated sequence, the following general enumerative question is 
open: 

How many elements of class C 0/ size n do not contain any k-nesting? 

For all of the above classes, this question is equivalent to counting the number of fc-noncrossing objects. 
The matching case was resolved by Chen et al. [6], giving rise to lovely formulas. The set partition case was 
addressed by Bousquet-Melou and Xin [4], who found explicit results in the case of fc = 3. Their method 
passes through Young tableaux and restricted lattice paths to deliver functional equations, which are then 
analyzed using the kernel method. They showed that the generating function of 3-noncrossing set partitions 
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is D-finitc i.e., it satisfies a linear differential equation with polynomial coefficients — , and hypothesized 

that this is not the case for larger k: 

Conjecture 1.1 ([4]). For every k > 3, the generating function of k-noncrossing set partitions is not 
D- finite. 

Mishna and Yen [13] determined functional equations for fc-nonnesting set partitions, and described a 
process for isolating coefficients, giving additional evidence for Conjecture 1.1. To date in the literature, 
there is only very limited enumerative information on /c- nonnesting permutations for A; > 3. The case 
fc = 2 is easy, since 2-nonnesting permutations are those that avoid the pattern 321, that is, those without 
decreasing subsequences of length 3, and thus they are counted by the Catalan numbers. Note that, in 
general, fc-nonnesting permutations are not closed under pattern containment: the permutation 564312 is 
3-nonnesting, but it contains the pattern 54312, which is not a 3-nonnesting permutation. 

1.1. Main results. Our main contribution is a generating tree approach which passes through neither 
Young tableaux nor lattice paths, and can be used to describe both fc-nonnesting set partitions and k- 
nonnesting permutations, for arbitrary k. The generating tree construction leads to a functional equation 
for the generating function which provides us with information about the corresponding series. We describe 
generic functional equations in a number of variables which grows with k. 

The key innovation in this study is a new class of structures that are essentially arc diagrams "under 
construction" , which we call open arc diagrams. These are arc diagrams in which we allow semi-arcs with 
a single end-point. Usual arc annotated sequences form the subclass of diagrams with no semi-arcs. Aside 
from nestings, it is relevant to consider future nestings in open arc diagrams. We describe how to identify 
both patterns in a generating tree construction. 

We also find an interesting connection between open arc diagrams related to 3-nonnesting set partitions 
and Baxter permutations. 

1.2. Plan of paper. We describe open arc diagrams and their generating trees in Section 2. In Section 3 
we give the first application of this construction to determine a set of functional equations to enumerate 3- 
nonnesting set partitions. In Section 4 we give the construction for k-nonnesting partitions for general k, 
and we also consider partitions that avoid a related pattern called an enhanced fc-nesting. 

In Section 5, using similar ideas and combining the descriptions for usual and enhanced nestings in set 
partitions, we construct a generating tree for fc-nonnesting permutations and obtain analogous functional 
equations. 

Finally, in Section 6 we discuss future research and possible extensions of our work in connection with 
the original motivating problem, the study of RNA secondary structures. 

2. Open arc diagrams: a generalization of set partitions 

2.1. Arc diagrams. An arc diagram of size n is an embedded graph with vertices 1 to n drawn in an 
increasing row. We take the convention of labelling our diagrams from left to right, so we can refer to left 
and right end-points of an arc. We first focus on the case of arc diagrams representing set partitions, which we 
call partition diagrams for short. In this case, the arcs are always drawn above the vertices, and the partition 
block {ai, • • • ) dj}, where ui < 02 < • • • < aj, is represented by the arcs (ai, 02), (02, 03), . . . , (aj_i, Oj). 
Figure 2 shows the diagram of a partition of {1, . . . , 9}. 

Figure 2. The arc diagram representation of the partition {1, 3, 5}{2}{4, 6}{7, 8, 9}. 

The vertices of a partition diagram can be of four types: fixed points, openers, closers and transitories. 
A fixed point has no incident edges, an opener has degree one and is the left end-point of an arc, a closer 
has degree one and is the right end-point of an arc, and a transitory vertex has degree two and is the right 
end-point of one arc and the left end-point of another. In Figure 2, the vertex labelled 2 is a fixed point; the 
vertices labelled 1, 4, and 7 are openers; vertices 5, 6, and 9 are closers; and 3 and 8 are transitory. 
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2.2. Open arc diagrams. Wc generalize partition diagrams by allowing two additional vertex types: semi- 
openers and semi-transitories. These vertices are incident to semi-arcs with no right end-point, which we 
also call open arcs. We call these generalized diagrams open partition diagrams. To ensure a standard 
representation, wc shall continue the semi-arcs to a vertical line to the right of vertex n, and retain their 
order, not allowing the semi-arcs to intersect. We denote the semi-arc with left end-point i by {i, *). Figure 3 
displays an example. We view open partition diagrams as ancestors to proper set partitions, in a process 
which incrementally adds vertices in numerical order, possibly closing semi-arcs and/or opening new ones. 
The open partition diagram tt in Figure 3 is an ancestor to the two set partitions represented in Figure 4, 
among infinitely many others. 




123456789 



Figure 3. An example of an open partition diagram, tt. 




123456789 10 11 12 123456789 10 11 12 13 



Figure 4. Two set partitions with tt as an ancestor. 

An open partition diagram could also be viewed as a set partition in which each block is coloured one of 
two colours: one for proper blocks, and another for blocks ending with a semi-arc. For example, tt above 
represents the bi-coloured partition {1, 3}, {5}, {7}, {2}, {4, 6}, {8, 9}. As such, it is easy to determine that 
the exponential generating function for open partition diagrams is e^^ 

2.3. Generating tree for open partition diagrams. To construct a generating tree, we describe how to 
create open partition diagrams of size n from one of size n — 1 . There are four different possibilities for the 
type of vertex n when it is added to an open partition diagram of size n — 1: 

(1) fixed point; 

(2) semi-opener; 

(3) semi-transitory (provided there is an available semi-arc); 

(4) closer (provided there is an available semi- arc). 

For example, the diagram in Figure 3 generates the 8 diagrams in Figure 5, which we call its children. 

The number of children that can be generated by adding a vertex to the diagram depends only upon the 
number of semi-arcs. Suppose a diagram has m semi-arcs (coming from either semi-opener or semi-transitory 
vertices). Here are the possible ways to generate its children, depending on the type of the vertex that is 
added: 

(1) fixed point: one child with m semi-arcs; 

(2) semi-opener: one child with m -f 1 semi-arcs; 

(3) semi-transitory: m children, each with m semi-arcs; 

(4) closer: m children, each with m — 1 semi-arcs. 

This sums to a total of 2m -t- 2 children for any diagram with m semi-arcs. We remark that, in the notation 
of [1] , this generating tree is described by the system 

[(2) : {2t) {2eY{2e + 2){2e - 2)^-1], 

where the label of a diagram equals twice its number of semi-arcs. From this generating tree, one can recover 
the exponential generating function e"^^ ~^ for open partition diagrams. 
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1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 910 



Figure 5. The open partition diagram for tt and its children. 

In Sections 3 and 4 wc will modify this generating tree in order to incorporate the fc-nonnesting constraint. 
We will use the notion of hiturc nestings, which we define next. 

2.4. Nestings and future nestings. Recall that a fc-nesting in a partition diagram is a set of k mutually 
nesting arcs, that is, arcs ■ ■ ■ , {ik,jk) such that 

ii < 12 < ■ ■ ■ < ik < jk < jk-i <■■■ <ji- 
To incorporate the nesting constraint to the generating tree, we need to generalize the notion of fc-nestings 
to open partition diagrams. We define a (regular) k-nesting in an open partition diagram to be a set of k 
mutually nesting (closed) arcs. To handle semi- arcs, we introduce the concept of a future k-nesting, which is 
a set of fc — 1 arcs that form a regular k — 1-nesting together with a single semi-arc whose left end-point is to 
the left of first end-point of the k — 1-nesting (see Figure 6). As its name suggests, any proper set partition 
that is a descendant of a diagram with a future fc-nesting necessarily contains a fc-nesting, because the semi- 
arc in the future fc-nesting eventually has to close. Conversely, every partition diagram with a fc-nesting 
has an ancestor with a future fc-nesting, namely one where all but the outermost arc in the fc-nesting have 
been closed. Note that having multiple semi-arcs above a fc — 1-nesting does not guarantee ^-nestings in its 
descendants for £ > k. For example, the three semi-arcs in Figure 3, together with the arc under all three of 
them, become a 4-nesting in the partition on left of Figure 4, but they become three separate 2-nestings in 
partition on the right. 




1 2 3 4 5 6 7 



Figure 6. An example of a future 4-nesting. 

We say that an open partition diagram is k-nonnesting if it contains neither regular nor future fc-nestings. 
To enumerate fc-nonnesting partitions, our strategy is to generate fc-nonnesting open partition diagrams. 
Of these diagrams, fc-nonnesting set partitions correspond to the ones that have no semi-arcs. As such, we 
are generating more objects than we require, but we can specialize the generating function for the more 
general class of objects to obtain the generating function for the subset of fc-nonnesting partitions. Mishna 
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and Yen [13] gave a construetion whieh generates only this class, but our construction has two important 
advantages over theirs: it can handle the enhanced case (see Section 4.2), and it can be extended to k- 
nonnesting permutations (see Section 5). 



We warm up with the case of 3-nonnesting set partitions, as it forms the template for the general case 
for both partitions and permutations. In Section 3.1 we describe the tree in the 3-nonnesting case, and in 
Section 3.2 we derive a functional equation for the corresponding generating function, which we manipulate 
in Section 3.3 using the kernel method to obtain a recurrence for its coefficients. 

3.1. Generating tree. Let us construct a generating tree for 3-nonnesting open partition diagrams as 
a subtree of the one in Section 2.3. To each 3-nonnesting open partition diagram, we associate a label 
containing two pieces of information: the number of semi-arcs in the diagram, and the number of semi-arcs 
which belong to a futm-e 2-nesting. The latter value is the number of semi-arcs above at least one closed arc. 
For example, in Figure 7, the arrows indicate the semi-arcs which are in future 2-nestings. Since among the 
four semi-arcs there are two with this property, we associate to this diagram the label [4, 2] . Now consider 
a child of this diagram where the added vertex 12 is a closer or semi-transitory. If the added vertex closed 
the semi-arc originating from vertex 7, then (7, 12), (8, 9) and (3, *) would form a future 3-nesting. This is 
precisely what we are trying to avoid. On the other hand, vertex 12 can close any of the other semi-arcs 
without creating a future 3-nesting. Additionally, no such nesting is created if the added vertex 12 is a fixed 
point or a semi-opener. 



Suppose now that tt is an arbitrary 3-nonnesting open partition diagram. To avoid creating a future 
3-nesting when we add a new vertex to tt, the only restriction is that if the added vertex is a closer or 

semi-transitory, it must not close a semi-arc of a future 2-nesting of tt, except if it is the very top semi-arc. 
Note also that since tt does not contain future 3-nestings, the addition of a vertex cannot create a regular 
3-nesting. 

Next we show how to do the bookkeeping on the labels to incorporate this restriction. A fixed point or 
a semi-opener added to a diagram tt has no effect on whether or not a semi-arc is in a future 2-nesting. 
Suppose now that a closer vertex is added to tt. If this vertex closes a semi-arc not belonging to a future 
2-nesting of tt, all the semi-arcs above it are now in a future 2-nesting, even if they were not in one of tt, and 
the semi-arcs below it are unaffected. If the new vertex closes a semi-arc belonging to a future 2-nesting of 
TT, then it must be the top semi-arc, and in this case, the number of semi-arcs is reduced by one, as is the 
number of semi-arcs in a 2-nesting. A semi-transitory vertex added to tt behaves exactly like a closer, except 
that a new semi-arc, not belonging to any future 2-nesting, is created. 

Applying these considerations to the example from Figure 7, the different types of vertices that can be 
added and the resulting labels are described below. 

(1) Fixed point: one child with label [4, 2]. 

(2) Semi-opener: one child with label [5, 2], as one semi-arc not contained in future 2-nestings is added. 

(3) Semi-transitory: the number of semi-arcs is preserved, and the number of those belonging to future 
2-ncstings depends on the semi-arc that is closed as follows. 

• (11,*), the three arcs above it become part of future 2-nestings, so the child has label [4,3]; 

• (10,*), the two ares above it remain part of future 2-nestings, so the child has label [4, 2]; 

• (3,*), the number of semi-arcs in future 2-nestings is reduced by one, so the child has label [4, 1]. 
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3. 3-NONNESTING SET PARTITIONS 




123456789 10 11 



Figure 7. An open partition diagram with label [4,2]. 



(4) Closer: the number of semi-arcs is reduced by one, but otherwise it is analogous to the semi-transitory 
case, hence the three labels are [3, 3], [3, 2], [3, 1]. 
Remark that the number of children and their labels depend only upon the label of the parent. Indeed, 
suppose the parent is a diagram with label Adding a fixed point produces a child with label 

while adding a semi-opener we get a child labelled + 1, j]. Assuming that i > 1, adding a closer yields 
i — j children with labels ranging from [z — 1, j] to [z — l,z — 1], corresponding to closing the semi-arcs not 
belonging to future 2- nestings, and, if j > 0, an additional child with label [i — l,j — 1], corresponding to 
closing the top semi-arc in a future 2-nesting. Adding a semi-transitory vertex has a similar effect to adding 
a closer, but the first part of the label in the children is i instead of z — 1. This description leads to the 
following theorem about the generating tree scheme, whose first few levels are shown in Figure 8. 

Theorem 3.1. Let II'^^^ be the set of 3-nonnesting open partition diagrams. To each diagram, associate 
the label ^(tt) = [i,j] if tt has i semi-arcs, j of which belong to some future 2-nesting. Then the numher of 
diagrams in n'^-' of size n is the number of nodes at level n in the generating tree with root label [0,0], and 
succession rule given by 

[i,j], (fixed point) 

[i + (semi-opener) 

+ 1]) • • • ) [^i * ~ 1]) if i>l (semi-transitory) 

+ «/i>l (closer) 

— 1], [i — l,j — 1]. if and j > (semi-transitory and 

closer when j > 0) 

The number of 3-nonnesting set partitions of {1, ... , n} is equal to the number of nodes with label [0, 0] at 
level n. 



(1) [^J] 



[0,0] 




[0,0] [1,0] [1,0] [2,0] [1,0] [0,0] 



Figure 8. Generating tree for 3-nonnesting open partition diagrams. The first few lev- 
els agree with the generating tree for all open partition diagrams, since the 3-nonnesting 
restriction only influences levels further down the tree. 



3.2. Functional equation. Here we translate the generating tree from Theorem 3.1 into a functional equa- 
tion for the ordinary generating function enumerating 3-nonnesting open partition diagrams by size and 
label. The generating function for 3-nonnesting set partitions will then be obtained by a simple evaluation. 

Consider the generating function A{u, v;z) = '^ Aij {n)u^v^ z"', where Aij (n) is the number of 3-nonnesting 
open partition diagrams at level n of the generating tree with label [i,j]. For the sake of simplicity, we 
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henceforth use A{u, v) to denote A{u, v; z). The root of the tree, the empty set partition, has label [0, 0] and 
translates to the term u°v°z° = 1. Next we consider the recursive step that generates each level from the 
previous one by adding vertices of one of four different types. 

Prom the addition of a fixed point or a semi-opener, that is, the first two lines of Equation (1), we get 

z{A{u, v) + uA{u, v)). 

Let us now consider the addition of a semi-transitory vertex. The addition of a closer is analogous 
substituting the term u^~^ for m*. If j > i, we have that Aij{n) = 0. The rules for adding a semi-transitory 
in Equation (1) translate to 

z ^ Ai,o(n)w*(l +v + v'^ + --- + v'-'^)z" if j = 0, 

i,n 

z ^ Aij{n)u\v^-^ + ■■■ + w'-^)^" if < i < i. 
Adding these, multiplying by z"', and using the formula for finite a geometric sum, we get 

ij,n i,j>l,n 

Combining the expressions for all four types of vertices, we obtain the following functional equation. 

Corollary 3.2. The generating function A{u, v) for ^-nonnesting open partition diagrams, with variables 
u and V marking values i and j in the label, respectively, and z marking number of vertices, satisfies the 
functional equation 



(2) 



X . X 1\ /A(u,v) - A{uv,l) A(u,v) - A{u,0)\\ 

Aiu, v) = l + z[il + u)Aiu, v)+[l + -) + ' ' \ ' ' ' j j 



3.3. Manipulating the equation. We use two variable transformations to simplify the functional equa- 
tion (2), in preparation for the application of the algebraic kernel method in order to extract coefficients 
Ao,o("), the number of set partitions without a 3-nesting. The method we follow for the analysis is parallel 
to the methods in [3, 13]. 

To get an expression for Aofi{n) in terms of the coefficients Aij{k), for fc < n — 1, we generate functional 

equations equivalent to Equation (2) , which are then taken in an alternating sum. The resulting cancellations 
yield a new functional equation from where we extract the sub-series AQfi{n)z^\ which is the generating 
function for 3-nonnesting set partitions. 

3.3.1. A simplifying change of variables. First, to remove the restriction that the exponent of u is always 
greater than or equal to the exponent of t;, we replace v hy t = v/u to get 

A{u,t) = J2 A,j{n)uHh'' = J2 Aijiny-^vh"", 

i.j.n T--j.TL 

renamed as A*{u, v) := ■ Aij (n)u^^hj^ z"^ . Then the various forms of A are transformed as follows: 

A{u,v) ^ A*{u,uv), A{uv,l) = A*{uv,uv), A{u,0) = A*{u,0). 
Equation (2) in terms of A* becomes 

X*/ X , /^/. X.*/ X 1\ / A*(u,uv) - A*(uv,uv) A*(u,uv) - A*(u,0) 

A*{u,uv) = 1 + z(il+ u)A*{u,uv) + (1 + -){ — ^— — — '- + — — 

\ \ u/ \ 1 — V V 

which, letting w = uv, simplifies to 

X . X.*/ X l\fA*{u,w)-A*{w,w) A*{u,w) - A*{u,0) 
A*{u,w) = l + zUl + u)A*{u, w) + (l + - j ( w + w 

\ u — w w 
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By collecting all the terms with A*{u,'w), we get the functional equation in its kernel form 

A*{w,w) A*{u,Oy 



K{u, w)A*{u,w) = l + z{l + u) 
where 



w — u w 



K{u,w) = l-z{l + u)[l+ ^ ^ 



u — w w 



3.3.2. A symmetrizing substitution and an alternating sum. To achieve symmetry in the kernel, we let u = 
I + w to get 

1 1 



K(l,w) = K(l + w,w) = l-z(l+w + l){l . , , 

\ I w 

which is symmetric in I and w. Now, we introduce a multiplicative factor M{1, w) — luP on both sides of 

A*{w,w) A*{l + w,Qy 



w 



K{l,w)A*{l + w,w) = l + z{l + l + w) 

to get 

(3) K{l,w)lw^A*{l + w,w) = Iw"^ + z{l + 1 + w) {-w'^A*{w,w) -lwA*{l + w,0)) . 
Note that the last term 

-z{l + 1 + w)lwA*{l + w, 0) 

is invariant when I and w are interchanged. Now we apply the algebraic kernel method where we take the 
orbit sum of Equation (3) over the symmetric group of order 2, ©2- 
The signed orbit sum of Equation (3) divided by K{l,w)M{l, w) is 

E a{lw'^A*{l + w,w)) 1 ■r^ a{lw'^) z{l + l + w) s--^ a{-w'^A*{w,w)) 
Next, we expand the orbit sum to get 

(4) A*{1 + w,w) - -A*{1 + w,l) = -^j^Al - -)+ ^ '''^ (^A*{1,1)-\a*{w,w)) . 

w K{l,w) w K{l,w) \w'' I J 

3.3.3. Series extraction. Our goal is to extract AQfi{n), the coefficient of u'^v'^z". This is equivalent to 
obtaining the series A{0, 0) = A*{0, 0). Let us define a linear operator £ for this purpose: 



1 ifi = j = 0; 
otherwise. 



When C is applied to Equation (4), the left hand side gives 

C (A*a + w,w) - -A*(l + w,l) 
\ w 



= ^ylo,o(n)2", 



because the second expression with l/w always contains terms with at least one I, thus C maps it to 0. 
Applying C to the right hand side of Equation (4) , we obtain 



i+j+k=n 

-.£|>'^''<?C,;:)C-:,;-:;..)))- 

Theorem 3.3. The number of ^-nonnesting set partitions o/ {1, . . . , n} is 

i+j+k=n 

The sequence defined by the first sum has exponential generating function e^^(l — z)~^ . Ideally, we would 
like to turn the latter two sums into a tighter recurrence solely in terms of Ao,o(fc), k < n. 

4. fc-NONNESTING SET PARTITIONS 

In Section 4.1, we generalize the construction of the generating tree and the functional equation from 
Section 3 to arbitrary k. In Section 4.2, we consider a variation of the notion of nestings, and derive 
analogous results for this case. In Section 4.3, we present enumerative data that can be obtained with our 
equations, and a surprising connection to Baxter permutations. 

4.1. Generating tree cind functional equation. It will be convenient to shift the index and consider 

k + 1-nonnesting partitions. Suppose that tt is an fc + 1-nonnesting open partition diagram. To avoid 
creating a future k + 1-nesting when adding a new vertex to tt, the only restriction is that, if the added 
vertex is a closer or semi-transitory, it is not allowed to close a semi-arc belonging a future fc-nesting of tt, 
unless it is the very top semi-arc. Note also that since tt does not contain future k + 1-nestings, the addition 
of a vertex cannot create a regular k + 1-nesting. 

In order to keep track of future fc-nestings, we also need to keep track of future j-nestings for j < k. 
We define the nesting index of a semi- arc to be the maximum j such that there is a j-nesting beneath it. 
Equivalently, the nesting index of a semi-arc is the largest j such that the semi-arc is in a future j + 1-nesting. 
We will keep track of the distribution of nesting indices on the semi-arcs, updating this distribution every 
time we add a vertex, and avoiding the appearance of future k + 1-nestings. 

To each k + 1-nonnesting open partition diagram tt we associate a label with k components £(7r) = 
[so, . . . , Sfe-i], where Sj is defined to be the number of semi-arcs of nesting index greater than or equal to i. 
Figure 9 contains an example. Remark that ,so is the total number of semi-arcs, and .Sfe_i is the number of 
semi-arcs in future /c-nestings. For k = 2, this labelling is consistent with Section 3.1. Furthermore, note 
that, by definition, so > si > ■ ■ ■ > Sk-i > 0. The label of the empty partition is [0,0, ...,0], since it 
contains no semi-arcs. 

As before, we can predict the labels of the children of a given node from its label alone, based on an 
analysis of the four types of nodes that we can add. When a semi-arc is closed, the effect on the labels is 
determined by the fact that the nesting index of those semi-arcs above it which had the same nesting index 
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Figure 9. An open partition diagram with label [5,4,2,1]. The nesting index of each 
semi-arc is labelled in italics. 

increases by one. This allows us to describe a succession rule for the generating tree, summarized in the 
following theorem, which generalizes Theorem 3.1. 

Theorem 4.1. Let he the set of k + 1-nonnesting open partition diagrams. To each diagram, associate 
the label £{'jt) = [sq, • • • , Sfc-i], where Si is the number of semi-arcs with nesting index > i. Then, the number 
of diagrams in H^*^) of size n is the number of nodes at level n in the generating tree with root label [0, 0, . . . , 0] , 

and succession rule given by 

[so, Sl, . . . , Sk-l] 

[so,si,...,Sk-i], (1) 

[.so + l,si,...,Sfe_i], (2) 

[sQ,si - 1, . . . ,Sj_i - l,?,Sj+i, . . . ,Sfc-i], for I < j < k ~ 1, and sj <i < sj-i - 1 (3) 

[sq — 1, Sl — 1, . . . , Sj^i — 1, i, Sj^i, . . . , Sfe_i], for 1 < j < k — 1, and Sj <i < Sj-i — 1 (4) 

[so,si - 1, . . . ,Sfe-i - 1], [so - l,.si - 1, . . . ,Sfc-i - 1], i/sfe_i>0. (5) 

Proof. The labels arise from adding the following kinds of vertices: 

(1) a fixed point; 

(2) a semi-opener; 

(3) a semi-transitory; 

(4) a closer; 

(5) a semi-transitory or a closer that closes the top semi-arc, if the diagram had a future fc-nesting. 

□ 

Corollary 4.2. The generating function for k + 1-nonnesting open partition diagrams, with variable Vi 
marking value Si in the label, and z marking number of vertices, denoted Q = Q{vo,Vi, . . . ,Vk-i) = 
Q{vo, vi, . . . , Vk-i; z), satisfies the functional equation 

Q = l + z{l^- vo) (q + (Q - Q{vo, vi,..., Vk-2, 0)) 

V VoVl...Vk-l 

fe-1 



y2 —71 ^{Q - <3(^0, • • • , Vj-2,Vj-iVj, 1, Vj+i,. . . , Vk-2,Vk-l))) . 



Setting Vo = 0, we obtain the generating function for k + 1-nonnesting partitions, Q{0,vi, . . . ,Vk-i;z). 
Note that this is a function of z only, because of the restriction so > si > ■ • ■ > Sfe_i > 0. The functional 
equation in Corollary 4.2, which specializes to Corollary 3.2 for /c = 2, is amenable to series generation, and 
it remains to be seen what can be extracted using a kernel method analysis. 

4.2. Enhanced nestings. Another relevant pattern in partition diagrams is the enhanced k-nesting. An 
enhanced fc-nesting is cither a fc-nesting, or a set of A; — 1 arcs {ii,ji), . . . ,{ik-i,jk-i) and a fixed point 
vertex ik (a singleton block in the corresponding partition) such that 

ii <i2 < ■■■ < ik-i <ik < jk-i < ■■■ < ji, 
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that is, a fc — 1-ncsting with a. fixcid point inside the innermost arc. With a comparable definition for enhanced 
A;-crossings, the symmetry between nesting and crossing patterns for different structures has been shown to 
hold in the enhanced version of the patterns as well. Set partitions are one such structure. Bousquet- 
Mclou and Xin [4] considered both enhanced and usual crossings in set partitions. Our construction that 
generates ^-nonnesting partitions can be easily modified to generate partitions with no enhanced fc-nesting. 
For this purpose, we define a future enhanced k-nesting as an enhanced fc — 1-nesting together with a semi-arc 
beginning to its left, and we let the enhanced nesting index of a semi-arc be the largest j such that it is in 
a future enhanced j + 1-nesting. The labels of the nodes in the generating tree now keep track of semi-arcs 
according to their enhanced nesting index. The only difference in the construction of the tree is that the 
addition of a fixed point to an open partition diagram tt can create future enhanced 2-nestings. Indeed, 
the semi-arcs that had enhanced nesting index in tt have enhanced nesting index 1 after the fixed point is 
added. This results in the following variation of Theorem 4.1. 

Theorem 4.3. Let n'*^-' be the set of open partition diagrams with neither enhanced k+l-nestings nor future 
enhanced k + 1-nestings. To each diagram, associate the label i{w) = [sq, • • • , Sk-i], where Si is the number 

of semi- arcs with enhanced nesting index > i. Then the number of diagrams in 11*^'^) of size n is the number 
of nodes at level n in the generating tree with root label [0,0,..., 0], and succession rule given by 

[so,Si,---,Sfe-i] 



Proof. The labels arise from adding the following kind of vertices: 

(1) a fixed point; 

(2) a semi-opener; 

(3) a semi-transitory; 

(4) a closer; 

(5) a semi-transitory or a closer that closes the top semi-arc, if the diagram had a future enhanced 



This succession rule can again be translated into a functional equation for the corresponding generating 
function. 

Corollary 4.4. The generating function for open partition diagrams with neither regular nor future enhanced 
k + 1-nestings, with variable Vi marking value Si in the label, and z marking number of vertices, denoted 
P = P{vo, vi, . . . , Vk-i) = P{vo,vi, . . . , Vk-i; z), satisfies the functional equation 



[so, So, S2, • 
[so + l,Sl, 
[so. Si - 1, 
[so - l,Si - 
[so,Si - 1, 



, Sfe-l], 
• , Sfe-l], 

. , Sj_i — 1, i, Sj+i, . . . , Sfe-i], for 1 < i < fc — 1, and sj < i < Sj-i — 1 
1, . . . , Sj-i — 1, i, Sj+i, . . . , Sfe-i]) for 1 < j < k — 1, and Sj <i < sj-i 
. ,Sfc_i - 1], [so - l,si - 1, . . . ,s/c_i - 1], ifsk-i > 0. 



1 



(1) 
(2) 
(3) 
(4) 
(5) 



fc-nesting. 



□ 



P=l + z\voP + 



1 + vo 



{P-P{vo,Vl,...,Vk-2M 



VqVi . . . Vk-1 



k-1 



l+Vo 



VQVl . ..Vj-l{l - Vj) 



{P - P{vo, . . . , Vj-2, Vj-iVj, 1, Vj+i, Vk-l)) 




Again, P(0, vi, . . . , Vk-i;z), which is a function of z only, is the generating function for partitions avoiding 
enhanced fc -|- 1-nestings. 
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4.3. Enumerative data, and a connection with Baxter permutations. Corollaries 4.2 and 4.4 allow 
us to generate data for the number of set partitions of size n avoiding k + 1-nestings and avoiding enhanced 
fc + l-nestings for small k and n. Tables 1 and 2 present the initial counting sequences, and relevant references 
to the On-line Encyclopedia of Integer Sequences [12]. We are able to generate more many terms than are 
listed. 



k + 1 
3 



OEIS 
A108304 

A108305 

A192126 

A192127 

A192128 



Initial terms 

1, 2, 5, 15, 52, 202, 859, 3930, 19095, 97566, 520257, 2877834, 16434105, 96505490, 580864901, 
3573876308, 22426075431, 143242527870, 929759705415, 6123822269373, 40877248201308 
1, 2, 5, 15, 52, 203, 877, 4139, 21119, 115495, 671969, 4132936, 26723063, 180775027, 1274056792, 
9320514343, 70548979894, 550945607475, 4427978077331, 36544023687590, 309088822019071 
1, 2, 5, 15, 52, 203, 877, 4140, 21147, 115974, 678530, 4212654, 27627153, 190624976, 1378972826, 
10425400681, 82139435907, 672674215928, 5712423473216, 50193986895328, 455436027242590 
1, 2, 5, 15, 52, 203, 877, 4140, 21147, 115975, 678570, 4213596, 27644383, 190897649, 1382919174, 
10479355676, 82850735298, 681840170501, 5828967784989, 51665915664913, 473990899143781 
1, 2, 5, 15, 52, 203, 877, 4140, 21147, 115975, 678570, 4213597, 27644437, 190899321, 1382958475, 
10480139391, 82864788832, 682074818390, 5832698911490, 51723290618772, 474853429890994 



Table 1 . Counting sequences for k + 1-nonnesting set partitions. 



k + 1 
3 



OEIS 
A108307 

A192855 

A192865 

A192866 

A192867 



Initial terms 

1, 2, 5, 15, 51, 191, 772, 3320, 15032, 71084, 348889, 1768483, 9220655, 49286863, 269346822, 
1501400222, 8519796094, 49133373040, 287544553912, 1705548000296, 10241669069576 
1, 2, 5, 15, 52, 203, 876, 4120, 20883, 113034, 648410, 3917021, 24785452, 163525976, 1120523114, 
7947399981, 58172358642, 438300848329, 3391585460591, 26898763482122, 218263920521938 
1, 2, 5, 15, 52, 203, 877, 4140, 21146, 115945, 678012, 4205209, 27531954, 189486817, 1365888674, 
10278272450, 80503198320, 654544093035, 5511256984436, 47950929125540, 430240226306346 
1, 2, 5, 15, 52, 203, 877, 4140, 21147, 115975, 678569, 4213555, 27643388, 190878823, 1382610179, 
10474709625, 82784673008, 680933897225, 5816811952612, 51505026270176, 471875801114626 
1, 2, 5, 15, 52, 203, 877, 4140, 21147, 115975, 678570, 4213597, 27644436, 190899266, 1382956734, 
10480097431, 82863928963, 682058946982, 5832425824171, 51718812364549, 474782378367618 



Table 2. Counting sequences for set partitions avoiding enhanced k + 1-nestings. 



In our computations, we remarked that the number of open partition diagrams on n vertices with neither 
regular nor future enhanced 3-nestings coincides with the number of Baxter permutations of size n for up to 
at least 200 terms. Specializing the functional equation from Corollary 4.4 gives 

B{u,v) = 1 

/ , B(u,v) -vB(uv,l) B(u,v) - B(u,0) B(u,v) - B(uv,l) B(u,v) - B(u,0)\ 

\ 1—V V u{l — V) uv J 
The initial series development is 

B{u, v) = 1 + {u+l)z + {uv + u'^ + 2u + 2)z^ + [v^u^ + 2vu^ + Auv + &u + + + h)z^ 

+ {u^v'^ + Iv^u^ + 2v'^u^ + l&uv + llvM^ + 3t;u^ + 20u -|- 12u^ + Au^ + u^ + 15)^^ + .... 
Evaluated at u = w = 1, it seems to coincide with the generating function for Baxter permutations: 
B{1, 1) = 1 -I- 2^ 6^;^ -I- 22z^ + 92z'^ + A22z^ + 2074^^ -|- 10754^;^ + 58202^^ -|- 326240^;'' + 1882960^;^° + . . . . 
This leads us to the following conjecture. 

Conjecture 4.5. LeA B„ be the set of open partition diagrams on n vertices with neither regular nor future 
enhanced 3-nestings. There exists a bijection between Bn and the set of Baxter permutations of length n+ 1. 
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The above sequence is also known to count plane bipolar orientations [2]. This raises the question of 
whether open partition diagrams avoiding regular and future enhanced A;-nestings are in bijection with some 
family of maps. The generating trees known for Baxter permutations and plane bipolar orientations are very 
different from the trees that we get for open partition diagrams. 



There are some similarities betwecin A':-nonnesting permutations and /c-nonnesting set partitions, yet none 
of the techniques that have been applied in the literature to the enumeration of ^-nonnesting set partitions [4, 
13] works for permutations. On the other hand, our method can be modified to take advantage of these 
similarities and deal with /c-nonnesting permutations as well. This gives rise to the first substantial set of 
enumerativc data on fc-nonnesting permutations. Previously, only data up to n = 12 was known, and its 
obtaining required extensive computation. 

First, we rcicall how to represent a permutation as an arc diagram. This representation was first used by 
Corteel [8], and in a modified form by Elizalde [11]. It is essentially a drawing of the cycle structure of the 
permutation. Given a £ 6„, the diagram of a has an arc between i and a{i) for each i from 1 to n, and the 
arc is drawn above the vertices (and is called an upper arc) if i < <T{i), and below the vertices (and is called 
a lower arc) if i > cT{i). We call such a representation a permutation diagram of size n. In this notation, a 
subset of arcs is a k-nesting if either 

(1) all k arcs are upper arcs and form an enhanced fc-nesting with the definition from Section 4.2 

(considering fixed points {i,i))', 

(2) all k arcs are lower arcs and form a fc- nesting with the definition from Section 1. 

We call the two above possibilities upper enhanced k-nestings and lower k-nestings, respectively. The reason 
behind the slight dissymmetry in the definition comes from the original paper of Corteel [8], where this 
was necessary for bijcctions between certain classes of permutations. Burrill et al. [5] maintained this 
dissymmetry. In any case, the construction we provide can be easily adapted to make a uniform treatment 
of upper and lower arcs. For simplicity, we do not draw an arc from i to itself when <j{i) = i, although such a 
fixed point contributes to enhanced fc-nestings of upper arcs. Figure 10 shows an example of a permutation 
diagram. 



Figure 10. The permutation a = (1 11 3) (2 6 4 5) (7 9) (8) (10) and its arc diagram represen- 
tation. This diagram has two 3-nestings: {(1, 11), (2, 6), (4, 5)} and {(1, 11), (7, 9), (8, 8)}. 

As in the case of set partitions, there is a related definition of fc-crossings in permutations. BiuTill et al. 
proved that fc-noncrossing and fc-nonnesting permutations are equinumerous. However, they present very 
limited enumerativc results, which we significantly improve upon here. 

We remark that the subset of upper arcs forms a set partition, and so does the subset of lower arcs. 
However, these two partitions are not independent. Their relationship is explicitly described in [5]. This 
interpretation as a pair of partitions allows us to extend the construction from Section 4 quite naturally. 

The vertices of a permutation diagram can be of five types: fixed points, openers, closers, upper transitories, 
and lower transitories. A fixed point has no incident edges, an opener (resp. closer) is the left (resp. right) 
end-point of an upper and a lower arc, and an upper (resp. lower) transitory vertex is the right end-point of 
an upper (resp. lower) arc and the left end-point of another. 



5. A;-NONNESTING PERMUTATIONS 
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5.1. Generating tree for open permutation diagrams. Similarly to what wo did for partitions, we 
consider a more general class of objects that we call open permutation diagrams, by allowing three additional 
vertex types: semi-openers, upper semi-transitories, and lower semi-transitories. A semi-opener is the left 
end-point of an upper and a lower semi-arc, and an upper (resp. lower) transitory vertex is the right end- 
point of an upper (resp. lower) arc and the right end-point of an upper (resp. lower) semi-arc. Note that in 
an open permutation diagram, the number of upper semi-arcs equals the number of lower semi-arcs. 

An open permutation diagram can also be viewed as a permutation where each cycle of length i can be 
coloured one of i -t- 1 possible colours: one to indicate that the cycle has no semi-arcs, and the other i for the 
possible choices of an arc in the cycles to be converted into two semi-arcs. It follows that the exponential 
generating function for open permutation diagrams is 




To create an open permutation diagram of size n from one of size n — 1, we add a vertex labelled n which 
can be of any of the five following types: 

(1) fixed point; 

(2) semi-opener; 

(3) upper semi-transitory (provided there is an available upper semi-arc); 

(4) lower semi-transitory (provided there is an available lower semi-arc); 

(5) closer (provided there are available upper and lower semi-arcs). 

We can easily make a generating tree for the class of open permutation diagrams by keeping track of the 
number £ of upper semi-arcs: 

[(0) : (2£) ^ {2£){2i + 2){2£f\2£ - 2)^']. 

To incorporate the nesting constraint to the tree, we define a notion of future nestings in open permutation 
diagrams. A future enhanced upper k-nesting is an upper enhanced k — 1-ncsting together with an upper 
semi-arc beginning to its left, and a future lower k-nesting is a lower k — 1-nesting together with a lower 
semi- arc beginning to its left. An example of each is in Figure 11. The enhanced nesting index of an upper 
semi-arc is the largest j such that the semi-arc is in a future enhanced upper j -\- 1-nesting. The nesting 
index of a lower semi-arc is the largest j such that the semi-arc is in a future lower j + 1-nesting. An open 
permutation diagram is called k-nonnesting if it has no regular or future enhanced upper fc-nestings, and no 
regular or future lower fc-nestings. 




Figure 11. An open permutation diagram. The upper arcs (7,12), (8,9) and the upper 
semi-arc (3, *) form a future enhanced upper 3-nesting. The lower arc (7, 10) and the lower 
semi-arc (3, *) form a future lower 2-nesting. The nesting index of each semi-arc is labelled 
in italics. 

Again, it will be convenient to shift the index and consider k + 1-nonnesting permutations. To each 
k -\- 1-nonnesting open permutation diagram, we associate a label consisting of a 3-tuple, [/i;r;s]. Here, h is 
the number of upper semi-arcs (and hence also the number of lower semi-arcs), r = [ri, . . . , rk-i] is a vector 
such that ri is the number of upper semi- arcs of enhanced nesting index greater than or equal to i, and 
s = [si, . . . , Sk-i] is a vector such that Si is the number of lower semi-arcs of nesting index greater than or 
equal to i. The label of the 4-nonnesting diagram in Figure 11 is [3; 1, 1; 1, 0]. 
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We note that in an open permutation diagram a, if we eonsider loops as fixed points, the upper (resp. 
lower) arcs and semi-arcs form an open partition diagram (resp. a^) on the vertices {1, . . . ,n}. If the 
label of (7 is [h; r; s] , then the label of cr+ as described in Theorem 4.1 is [h, r] , and the label of cr+ as described 
in Theorem 4.3 is [h, s]. In particular, /i > ri > ■ • ■ > rk-i > and h > si > ■ ■ ■ > Sk-i > 0. 

5.2. 3-nonnesting permutations. Again, the case of 3-nonnesting permutations is sufficiently insightful 
for the general method without being overly complicated. In this section we describe the generating tree 
for 3-nonnesting open permutation diagrams, and we determine a functional equation for the generating 
function. 

5.2.1. Generating tree. The label of a 3-nonnesting open permutation diagram is [h,ri,si] (we use commas 
instead of semicolons in this section). Here, 2h is the total number of semi-arcs, ri is the number of upper 
semi-arcs that belong to a future enhanced upper 2-nesting, and Si is the number of lower semi-arcs that 
belong to a future lower 2-nesting. An example of the labelling is given in Figure 12, where the numbers on 
the vertices are omitted. The arrows indicate semi-arcs that are in future 2-nestings: two upper semi-arcs 
and one lower semi-arc, hence the label of the diagram is [4, 2, 1]. 



Figure 12. An arc diagram with label [4, 2, 1]. 

To predict the labels of the children of a 3-nonnesting open permutation diagram, we consider the different 
types of vertices that can be added. To avoid future 3-nestings, we are not allowed to add a closer or semi- 
transitory vertex that closes a semi-arc belonging to a future enhanced upper 2-nesting or a future lower 
2-nesting, unless it is an outermost semi-arc, i.e., the top upper semi-arc or the bottom lower semi-arc. As an 
example, the labels of the children of the diagram in Figure 12 are generated by adding vertices of different 
types as follows. 

(1) Fixed point: one child with label [4,4, 1], since the upper semi-arcs belong now to future enhanced 

upper 2-nestings. 

(2) Semi-opener: one child with label [5, 2, 1]. 

(3) Upper semi-transitory: closing the upper semi-arcs that are not in future enhanced 2-nestings gives 
the labels [4,2,1] and [4,3,1]; closing the top semi-arc (the only one in a 2-nesting that we are 
allowed to close) removes one future upper 2-nesting, giving the label [4, 1. 1]. 

(4) Lower semi-transitory: all lower semi-arcs can be closed, and the four resulting labels are [4,2,0], 
[4,2,1], [4,2,2] and [4,2,3]. 

(5) Closer: we simultaneously and independently close an upper and a lower semi-arc, among those that 
we are allowed to close. There are three choices for the former and four for the latter, giving twelve 
children with labels [3,1,0], [3,1,2], [3,1,1], [3,1,3], [3,2,0], [3,2,2], [3,2,1], [3,2,3], [3,3,0], [3,3,2], 
[3,3,1], [3,3,3]. 

The succession rule for the generating tree is described in Theorem 5.1. Another example is drawn in 
Figure 13, and the first few levels of the tree appear in Figure 14. 

Theorem 5.1. Let S^^^ be the set of 3-nonnesting open perm,uta,tion diagrams. To each diagram, a associate 
the label £ (a) — [h,r,s], where 2h is the total number of semi- arcs, r is the number of semi- arcs in a future 
enhanced upper 2-nesting and, s is the number of semi-arcs in a future lower 2-nesting. Then, the number 
of diagrams in S^^^ of size n is the number of nodes at level n in the generating tree with root label [0, 0, 0], 
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[2,2,0] [3,0,0] [2,1,0] [2,0,1] 

[2,0,0] [2,0,0] [1,1,1] [1,1,0] [1,0,0] [1,0,1] 
Figure 13. A 3-nonnesting open permutation diagram and its children. 

and succession rule given by 

[h, h, s] (1) 

[h+l,r,s] (2) 

[/i, r, s]— >■ [h,i,s\ /or niaxjO, r — 1} < z < /i - 1 (3) 

[h, T,j] for max{0, s — 1} < j < /i — 1 (i) 

[h — 1, i, j] for max{0, r — l}<«</i — 1 and max{0, s — 1} < J < ^ — 1. (5) 

The number of 3-nonnesting permutations of size n is equal to the number of nodes with label [0, 0, 0] at the 
n-th level of this generating tree. 

Proof. The labels correspond to the addition of the following nodes to a diagram a: 

(1) a fixed point, which results in all the upper semi-arcs becoming part of future enhanced 2-nestings; 

(2) a semi-opener, which produces a new upper semi-arc and a new lower one, neither of which is in a 
future 2-nesting; 

(3) an upper semi-transitory closing a semi-arc not belonging to a future enhanced upper 2-nesting or, 
if r > 0, possibly closing the top semi-arc; 

(4) a lower semi-transitory closing a semi-arc not belonging to a future lower 2-nesting or, if s > 0, 
possibly closing the bottom semi-arc; 

(5) a closer, which can close any combination of an upper and a lower semi-arc among those allowed to 
close in parts (3) and (4). 

□ 

5.2.2. Functional equation. We translate the generating tree from Theorem 5.1 into a functional equation 
for the multivariate generating function F{u,v,w) = J2-^h,r,s{n-)u'^v'^w^z'^ where Fh,r,s{'n) is the number 
of open permutation arc diagrams at level n with label [h,r,s]. The coefficient i^o,o,o(?^) is the number of 
3-nonnesting permutations of {1, 2, . . . , n}. 

The translation process to derive a functional equation is analogous to the set partition case. We consider 
the five types of vertices in turn to analyze their contribution. The form of the functional equation is 

F{u, V,w) = 1 + + 'I'2 + *3 + *4 + *5). 

where is the contribution for adding a vertex of type (i), which we compute next. 

(1) Fixed point. Note that case (1) in the succession rule can alternatively be included by extending the 
range of i in case (3) to include h. Thus, it will be simpler to compute ^i + ^s in item (3) below. 

(2) Opener. "5^2 = uF{u,v,w). 
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[0,0,0] 




[0,0,0] [1,0,0] [1,1,0] [2,0,0] [1,0,0] [1,0,0] [0,0,0] 



I II II /B III II II 
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Figure 14. Generating tree for 3-nonnesting open permutation diagrams. The first few 
levels coincide with those of the tree for all open permutation diagrams. 

(3) Upper transitory and fixed point. *i + *3 = + F{u,v,w)-Fiu,o,n,) ^ ^^^^^ ^^.^^ 
formula for a finite geometric sum in the expressions below: 

z J2 Fh,o,n{n)u''il +v + v^ + ... + v^)w'z'' if 3 = 0, 

h^s.n 

z Fh,r,e{nWh{v'-^ + ...+ v'')w'z'' if < j < i 

(4) Lower transitory. *4 = ^^"■"'"ll^^""''''''^ + 

(5) Closer. The addition of a closer to a diagram with label [h, r, s\ contributes 

F?,,^,,(n)u''-i(^;'"'"{'-i'°> + . . . + ^;^-i)(u;max{^-i,o} + . . . + 

h,n 

which can be simplified using finite geometric sum formulas, and separating the case when r = or 
s = 0: 

^ _ F{u, V, w) — F{uw, l,w) - F{uw, v, 1) + F{uvw, 1, 1) 
^ u{l — v){l — w) 

_^ F{u, V, w) - F{u, 0, w) - F{uw, v, 1) + F{uw, 0, 1) 

uv{l — w) 

_^ F{u, V, w) - F{u, V, 0) - F{uv, l,w)+ F{uv, 1, 0) 
^^(l — v) 

F{u, V, w) - F{u, 0, w) - F{u, V, 0) + F{u, 0, 0) 
uvw 

Adding all five contributions, we get the following corollary. 

Corollary 5.2. The multivariate generating function for 3-nonnesting open partition diagrams, denoted 

F{u,v,w;z)= Y, Fh,r,s{'"')u^v'^w^z"', 

h,r.s,n 
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where Fh,r,s {n) is the number of diagrams of size n with label [h, r, s] , satisfies the functional equation 



F{u, v,w) — 1 + zl uF{u, V, w) 



F{u,v,w) — v{uv,l,w) F{u,v,w) — F[u,0,w) F[u,v,w) — F{uw,v,l) 
1 — V V 1 — w 

^ F{u, V, w) — F{u, V, 0) ^ F{u, V, w) — F{uw, 1, w) — F{uw, v, 1) + F{uvw, 1, 1) 
w u(l — ?;)(! — w) 

F{u, V, w) - F{u, 0, w) - F{uw, v, 1) + F{uw, 0, 1) 



+ 
+ 



uv{l — w) 

F{u, V, w) - F{u, V, 0) - F{uv, 1, w) + F{uv, 1, 0) 



^^(l — v) 

F{u, V, w) - F{u, 0, w) - F{u, V, 0) + F{u, 0, 0) 



+ 



uvw 



We have found this equation useful to generate terms in the sequence, but we have been unable to solve 
it, or to find an explicit expression for Fofifi{n), the number of 3-nonnesting permutations. 

5.3. The general case: k + 1-nonnesting permutations. The construction in Section 5.2 can easily be 
generalized to this case. Recall that to each k + 1-nonnosting open permutation diagram we assign a label 
[h: r; s] = [h; ri, r2, . . . , rk-i; si, S2, ■ ■ ■ , Sfc-i]- To describe the succession rule of the corresponding generating 
tree, we think of [h, r] as the label of the upper set partition, where we consider enhanced nestings, and of 
[h, s] as the label of the lower set partition, where we consider usual nestings. We use r — 1 as a shorthand 
for n — 1, r2 — 1, . . . , Tfe-i — 1, and similarly for s — 1. When the parameters ro and so are used below in 
(36), (46), etc., they are defined to be equal to h. 

Theorem 5.3. Let E^*^) be the set of k + 1-nonnesting open permutation diagrams. To each diagram a 
associate the label £{cr) = [h; r; s] = [h; ri, r2, . . . , "Tfe-i; Si, S2, . . . , s^-i], where 2h is the number of semi-arcs, 
and ri (resp. Si) is the number of open upper (resp. lower) semi-arcs of enhanced nesting index (resp. 
nesting index) greater than or equal to i. Then the number of diagrams in S^*^^ of size n is the number of 
nodes at level n in the generating tree with root label [0;0;0], and succession rule given by 

[h; r; s] — > 

[h;h,r2,...,rk-i;s\, (1) 

[/i + l;r;s], (2) 

[/i;r- l;s], if rk-i > 1, (3a) 

[h;ri — 1, . . . . rj_i — 1, r^+i, . . . ,rfe_i; s], for 1 < j < k — 1 and rj<i< rj-i — 1, (3b) 

[h;r;s- 1], if Sk-i > 1, (4<i) 

[h; r; si — 1, . . . , Sj_i — 1, i, s^+i, . . . , Sfc-i], for 1 < J < k — 1 and Sj < i < Sj-i — 1, (4b) 

[/i — 1; r — 1; s — 1], if rk-i > 1 and Sk-i > 1 (5a) 

[/i - l;r - 1; si - 1, . . . ,Sj_i - s^+i, . . . , Sfe-i], 

*/ rk-i > 1, for I < J < k — 1 and Sj < i < Sj-i — 1 (5b) 

[h- l;ri - l,...,rj_i - 1, i, r^+i, . . . , r^-i; s - 1], 

if Sk-i > 1, for 1 < j < k — 1 and rj <i < vj-i — 1 (5c) 

[h - l;ri - 1, . . . , rj_i - l,i,rj+i, . . . ,rk-i;si - 1, ... , Sj_i - s^+i, . . . , Sfe-i], 

for 1 < J < k — 1 and Sj < i < Sj-i — 1, and for 1 < j < k — 1 and rj < i < rj-i — 1. (5d) 

Proof. The labels correspond to the addition of the following nodes to a diagram a: 
(1) a fixed point; 
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(2) a somi-opcncr: 

(3a) an upper semi-transitory closing the top semi-arc, if a has a future enhanced upper k — 1-nesting; 
(36) an upper semi-transitory; 

(4a) a lower semi-transitory closing the bottom semi-arc, if a has a future lower k — 1-nesting; 
(46) a lower semi-transitory; 

(5a) a closer that closes both the top and the bottom semi-arcs, if a has both a future enhanced upper 

k — 1-nesting and a future lower k — 1-nesting; 
(56) a closer that closes the top semi-arc and a lower semi-arc that is not the bottom one, if a has a 

future enhanced upper k — 1-nesting; 
(5c) a closer that closes the bottom semi-arc and an upper semi-arc that is not the top one, if a has a 

future lower k — 1-nesting; 
(5rf) a closer that closes an upper and a lower semi-arc, neither of which is an outermost one. 

□ 

We have also translated the succession rules of the generating tree for k + 1-nonnesting open permutation 
diagrams into a functional equation for the corresponding generating function, which has 2k variables. We 
omit it here due to space constraints. 

Initial terms 

1, 2, 6, 24, 118, 675, 4333, 30464, 230615, 1856336, 15738672, 139509303, 1285276242, 
12248071935, 120255584181, 1212503440774, 12519867688928, 132079067871313 
1, 2, 6, 24, 120, 720, 5034, 40087, 356942, 3500551, 37343168, 428886219, 5257753614, 
68306562647, 934747457369, 13404687958473, 200554264435218, 3118638648191005 
1, 2, 6, 24, 120, 720, 5040, 40320, 362856, 3627385, 39864333, 477407104, 6183182389, 
86033729930, 1278515941177, 20185987771091 

1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800, 39916680, 478991641, 6226516930, 
87157924751, 1306945300264 

Table 3. Counting sequences for k + 1-nonnesting permutations. 

5.4. Enumerative data. We used the gfun package of Maple (version 3.53) to try to fit the counting 
sequences for fc-nonncsting permutations (with 3 < fc < 6) into a differential equation using 80 terms, with 
no success. Thus, we make the following conjecture. 

Conjecture 5.4. The ordinary generating function for k-nonnesting permutations is not D-finite for any 
k>2. 

6. Perspectives 

Let us finish by describing some possible future directions of research extending our work. Having found 
functional equations for the generating functions of partitions and permutations avoiding fc-nestings (which 
are equinumerous to those avoiding fc-crossings), a natural extension would be to describe the distribu- 
tion of the number of fc-crossings and fc-ncstings on partitions and permutations. More generally, given 
ii, ?2, • • • , *r, .?i-.72, ■ • ■ - .Jr, what is the number of partitions (rcsp. permutations) of size n with ik fc-crossings 
and ^-nestings for 1 < fc, < r? Is this number the same if the words "crossings" and "nestings" are 
switched? A reasonable first step towards describing this distribution would be to study the number of £- 
nestings (for £ < fc) in fc-nonncsting partitions (rcsp. permutations). Perhaps probabilistic arguments could 
be used to obtain information about the expected number of ^-nestings. Also, it may be worth investigating 
whcthcir a notion of future fc-crossings can be used to translate the fc-nonc;rossing condition on open parti- 
tion diagrams into certain restrictions on some appropriately defined labels, from where a generating tree for 
fc-noncrossing partitions could be constructed. This might allow us to impose simultaneously noncrossing 
and nonnesting restrictions. 

In a different direction, we expect that our generating trees and functional equations can be used to code 
random generation schemes for fc-nonnesting partitions and permutations. Finally, it would be interesting 
to generalize our open diagram construction to study fc-nestings in other combinatorial structures, such as 
tangled diagrams and other objects used to predict RNA secondary structures. 
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